Conventionally, a pattern formed on a resist layer on the surface of a semiconductor wafer or liquid crystal substrate (hereinafter referred to as “sample”) is inspected for defects in manufacturing steps of a semiconductor circuit element or liquid crystal display element. For example, there is disclosed an inspection method which adjusts a polarization state of light from a light source and intensities of 0-th order and high-order diffracted light beams that form optical images and compares the images of the sample surface (e.g., see Patent Document 1). For such an inspection, Critical Dimension SEM (critical dimension scanning electron microscope, hereinafter referred to as “CD-SEM”) is used.